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Abstract 

We propose a network of oscillators to retrieve given patterns in which the os- 
cillators keep a fixed phase relationship with one another. In this description, 
the phase and the amplitude of the oscillators can be regarded as the timing 
and the strength of the neuronal spikes, respectively. Using the amplitudes for 
encoding, we enable the network to realize not only oscillatory states but also 
non-firing states. In addition, it is shown that under suitable conditions the 
system has a Lyapunov function ensuring a stable retrieval process. Finally, 
the associative memory capability of the network is demonstrated numerically. 
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Although the past decade has seen considerable advances in studies of neural networks, 
recent research seems to reveal the limitation of the current network models, composed of 
McCulloch-Pitts units or modifications of these. Provided that we will study the steady 
states of the network, these units are valid for modeling of actual neurons as a first approxi- 
mation. In fact, many fruitful results have been reported by using these units Jl[. However, 
when it comes to treating the dynamical behavior of the network, these models may fail to 
capture the essence of the dynamics. This is because these models ignore much of the de- 
tailed behavior of real neurons, such as the timing of neuronal firing and internal dynamics 
|^-^]. In some cases, such behavior seems to play a significant role in neuronal systems. 

For example, several recent experiments suggest that the temporal coherence of neuronal 
activity may contribute to segmentation of visual scenes ||. In this mechanism, synchro- 
nization of pulses of neuronal responses plays a crucial role in binding the visual features 
belonging to the same object. In other words, the phase relationships among the pulses 
become essential for information processing. Central pattern generators provide another 
good example. It is widely believed that a well-identified group of neurons controls rhyth- 
mic behavior of animals, i.e. locomotion, swimming, breathing and so on. Such a neuronal 
group is often called a central pattern generator. Particularly in case of an animal loco- 
motion, it is known that there are generally several distinct patterns of leg movements || . 
These patterns are classified by the relative phase relationships among the legs. Therefore, 
it is obvious that the corresponding neuronal system can generate several firing patterns for 
which there are different phase relationships among the neuronal pulses. 

To construct a theoretical model of the above systems, we need to describe the tempo- 
ral features of neuronal activities, such as synchronization and phase locking. However, a 
McCulloch-Pitts description is based on the assumption that information is encoded only 
by the averaged activities of the neurons. Therefore, such a description is too crude to 
represent temporal features of the firing states naturally. For example, if information is 
encoded by the relative phases of the spikes, this situation cannot be described naturally 
by the McCulloch-Pitts units. Hence, we need to construct another model which provides a 
suitable framework to grasp such temporal aspect of real neural networks. At the same time, 
it is required that such a model be simple enough to be mathematically tractable. For this 
purpose, a model in which neuronal activities take the form of oscillators is an attractive 
candidate [0-0. Using this idea, we can treat the timing of neuronal firings as the phase of 
the oscillators naturally. This is because the firing state can be treated as a limit cycle of a 
dynamical system where the time of emission of a pulse is indicated by the position on the 
limit cycle, namely, the phase <fi. Moreover, it is desirable that the non-firing state can also 
be expressed by the model. It is natural for the amplitude r of the limit cycle to be thought 
of as the strength of the pulse. Pursuing this idea, the non-firing state can be represented 
by the zero amplitude of the oscillator. For the neuron model, therefore, we will adopt an 
oscillatory unit whose state is determined by both the phase and the amplitude. For conve- 
nience of expression, we denote the state of the i-th neuron by the complex variable Wi with 
amplitude and phase fa. In this description, a non- firing state of the neuron corresponds 
to Wi = 0, while a firing state corresponds to Wi = exp(if2ji) (fli is the frequency of the 
i-th firing neuron.) 

Now we will construct a neural network model to retrieve phase information. Let us 
consider a network of N neurons whose dynamics are governed by the following system of 
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coupled oscillators 



dW- ~ N 

_i = viWi^+k^CijWj - W t ), (1) 
at j=1 

where the real variable k represents the total coupling strength, the complex variable CV, 
represents the effect of the interaction between the j-th and z-th neurons, and W% denotes 
the complex conjugate of W^ The function v(Wi, Wi) should be chosen so that the system 
can exhibit limit cycle behavior in the absence of the coupling term. In addition, we assume 
that the system ([!]) is invariant under uniform phase translation 

Wi->Wiexp(i0o), (2) 

where 0o is an arbitrary real constant. This requirement arises from the fact that infor- 
mation is encoded not by the absolute time but by the relative time of neuronal spikes. 
Correspondingly, the relative phase relationships among the oscillators are relevant to en- 
coding the information (not the absolute phase values). In this paper, for v(Wi, Wi) we will 
examine two types of functions satisfying the above two requirements. 

We here define notation for later discussion. Let £f (p — 1, . . . ,p) be a set of patterns to 
be memorized, where p is the total number of the patterns. We should remark that, owing 
to the invariance (H), all patterns generated by the uniform phase translation £f exp(i0 o ) 
represent the same pattern as £f . We also define p overlaps M M as the projections of the 
current state on the p embedded patterns: 



Mn = — 

N 



N 

i=i 



(3) 



At first, we consider the situation that all neurons are in the firing state. Stored patterns 
can then be characterized by the parameters 6*f (£f = exp(i0f)). Due to (Q), the effective 
number of degrees of freedom of these parameters is N — 1 for one pattern. As a simple 
choice, we consider the following dynamics of the first model 



,/U ' (1 + iQi)Wi - (1 + ic^WtfWi + k(J2 C i3 W j - Wi), (4) 

i=i 



dt 



where Qi can be considered as natural frequencies of neuronal spikes in the absence of 
external inputs, k controls the total strength of the coupling, and c is a real parameter. 
The system (f|) without the coupling terms is often referred as the Stuart-Landau equation. 
In general, it is well known that the Stuart-Landau equation is derived from a general 
ordinary differential equation near a Hopf bifurcation point. Therefore, this choice seems to 
be natural for oscillatory systems. For simplicity, we assume that c = and Oj = Q. Under 
this assumption, we can eliminate Q by the transformation Wi Wiexp(iQt). After simple 
calculations, therefore, (HT) reduces to 



dW N 

' Wi - \W t \ 2 Wi + k(J2 CijWj - Wi). (5) 



dt 
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Next we will show how given patterns are embedded in the network of oscillators, that 
is, how to determine the synaptic efficacies. At first, let us consider the condition that 
the equation (|5|) has the solution W{ = £f. Substituting Wi = £f into (|5|) and using 
|£f | = | exp(i6*f )| = 1, we obtain J2f=i — £f- This condition can be rewritten in matrix 
form as 

CP = P, (6) 

where the N x P matrix P is defined by Pij = £f . Provided that the total number p of the 
memorized patterns is smaller than N, the synaptic efficacies satisfying the condition (^) 
are then given by 

C = PP t + B(I-PP t ), (7) 

where P^ denotes the pseudo-inverse of P, I denotes the unit matrix, and B is an arbitrary 
matrix PD|JTT|] . If the patterns are linearly independent, the pseudo-inverse matrix pt is 

given by pt = 1/N (l/ArP T P) P T , where P and P T denote the complex conjugate and 
the transpose of P, respectively. Moreover, assuming B = O for simplicity, we obtain 

^^EE^ 1 )^, (8a) 
JV k=i i=i 



1 N - 
iv fc=l 

with = exp(i^). It is easily found that the synaptic matrix (Q) is Hermitian, that is, 

Cij = Cji. It is worth noting that if the patterns are ortho normal the synaptic prescription 

(§) recovers the generalized Hebbian rule = l/NJ2k=i^i^j- 

There are two trivial cases in this model. For p — 1, the synaptic efficacies are given by 
= 1/N exp(i9l —iOj). In this case, applying the transformation Wi exp(— i9j) — ► Wi, the 

dynamics of (|) reduce to 

dW- 1 N 

~^ = Wi- \WtfWi + k(— J2 Wj - Wi). (9) 



This equation has recently been studied by several authors fl2| , |T3|| . It is thought that this 



situation corresponds to the Mattis state in spin glass models. On the other hand, for p = N 
we obtain = 5ij. This implies that the couplings between the oscillators vanish and the 
dynamics of the network reduces to that of independent oscillators. 

We will show that the system (f|) reduces to a simpler set of phase oscillators under the 
conditions that the coupling is weak and c = 0. In the limit of weak coupling k ~ 0(e), 
the deviation of the amplitude from r = 1.0 may be neglected at first order. Inserting 
Wi = rjexp(i0j) into (|5|), this approximation immediately yields 

<fii = Qi + k \Qj\ sin(0 i -<f>i + aiij), (10) 
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where the a^- are defined by = \Cij\ exp(iaiij) . Since the synaptic matrix is Hermitian, 
we get 

\Cij\ = \Cij\ and oty = -a^. (11) 

Note that acij = — ct/i leads an = 0. As a result, self-coupling terms do not contribute to the 
dynamics of the network. 

In the previous model (§), all neurons are assumed to exhibit periodic firing states. 
However, such an assumption is not realistic because in a real system, for any given patterns, 
some neurons will be in a resting state. In the second model, we consider another type of 
dynamics where the amplitude plays a crucial role in representing non-firing state. Let us 
first consider the dynamics of a single neuron. To realize a stable non-firing state in retrieval 
patterns, it is desirable that the single neuron is able to exhibit both a periodic firing state 
and a non-firing state. To satisfy this requirement, we choose v(Wi, Wi) so that the dynamics 
of a single neuron can have both a limit cycle Wi = exp(zf2jt) and a stationary fixed point 
Wi = 0. As a simple choice, we employ v(W h W-) = (-1 + iQ^Wi + A\Wi\ 2 Wi - 3|Wi| 4 Wi. 
As is the case of the first model, assuming that the natural frequencies f2j are identical to 
Q, we can eliminate Q. Consequently, the dynamics of the network are described by the 
following 

Wi = -Wi + 4\W t \ 2 W t - 3|W f | 4 Wi + k {lt C ii W 3 ~ ■ ( 12 ) 

We next address the question of how to make the synaptic connections for patterns which 
include the possibility of non-firing states. The patterns are then defined by 

tn — / ex P(^) f° r firing state , , 

% 1 for non-firing state 

Putting the above patterns into ([12]) and using |£f | = 1 or 0, we obtain the same condition 
for the synaptic connections, CP = P, as in the first model. Therefore, if p < N and the 
patterns are linearly independent, we can apply the same procedure as in the first model. As 
a result, the synaptic matrix is given by the same prescription (|^). Note that the condition 
Cij = Cjt is kept even if the memorized patterns include some non-firing states. 

Above, it was shown that we can set the connections so as to make the pattern to be 
memorized the solution of the dynamical equation. To recall the embedded patterns from 
noisy ones, however, it is required that such solutions are the attractors of the dynamics. 
The existence of such asymptotic behavior is guaranteed by the existence of a Lyapunov 
function. In this section, we will show that in our models a Lyapunov function exists if the 
synaptic matrix is Hermitian and Qi = Q. The models which we have discussed so far can 
be written in the form 



A' 



dt 
with 



<m ' viWi^+k^djWj-Wi], (14) 
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v(Wi, Wy 



Wi - \Wi\ 2 Wi 

-W + 4|W| 2 W-3|Wl 4 W 



(15) 



Let us introduce a function V(Wi, Wi) by the following definition 



V{W t ,Wy 



dV{Wj, Wj) 
dW 



and v(Wi,W i: 



dVjW^W) 
dWi 



(16) 



where we have regarded Wi and Wi as independent variables. For example, the corresponding 
functions V(Wi, Wi) for flT5|) are given by 



|W| 2 -2|W| 4 + |W| 6 . 



(17) 



Provided that such a function V(W», Wi) exists, the Lyapunov function is given by 

N i N N N 

l(w u w) = J2 v(Wi, Wi)--J2J2 {CtjWiWj + c i3 WiW 3 ) + k J2 m\ 2 . 



(18) 



i=l j=l 



It is easily proved that L only decreases under the dynamics of (fM|). Using the function 
V(Wi, Wi), we can rewrite the model (14) in the from 



dWi 



dt \dWiJ 
From these relations, we immediately get 



dL \ , dW 4 
— ' and — — 
dt 



' dL 
dWi 



dL (dl^dW^ dL dW 

~dt ~ ^ 1 " 



<9W aw, dt 



N 

E 



2V 



< 0. 



(19) 



(20) 



Therefore, L can only decrease as a function of time. When the network converges to the 
memorized state Wj = L does not vary with time. We should remark that L is invariant 
under the uniform phase translation (0). 

To confirm the ability of the network, we carried out some numerical simula- 
tions. Among these simulations, we present one typical result. In this simulation, 
we used a network of 50 oscillators whose dynamics are govern by the equations (|12"D. 
The reason for using the second model is that we want to demonstrate the abil- 
ity of the network to retrieve phase patterns which include non-firing states. In all, 
eight patterns were stored by means of the synaptic prescription (||). The first pat- 
tern was ^ = (1,1,1,1,0,0,1,1,1,1, e' l ^~, e l ~£~ , e l ~5~ , e l ~£~ , 0, 0, e 1 "?, e i_ r, e*^", e*^", • ■ • , e 1 "?, 
e l ir, e l ~£~ , e l ~^ , 0, 0, e 1 ^, e l ~£~ , e*T", e l ~^). The remaining seven patterns £f (p = 2 ~ 8) were 
generated at random. That is, taking the form £f = A% exp(i#f ), 0f were chosen at random 
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from a uniform distribution between and 27r, and Af were independent random variables 
obeying the distribution 

A v = { 1 with probability ~ , , 

1 1 with probability | 

Figure 1 shows one typical temporal pattern of the retrieval process in which the parameter 
values are k = 1.0, Q = 2tt. Although Q make no contribution to the dynamics of the 
network, to express the phase relationships visually, the pattern in Figure 1 is illustrated in 
the rotating frame. In this context, the black horizontal bars represent active phases defined 
by RelVi > 0.5. The network was initially given the noisy pattern Q. Obviously, both 
the amplitudes and the phases were corrected dynamically. In the same simulation, time 
development of overlaps is shown in Figure 2. It is clearly found that the network succeeds 
in retrieving one of eight memorized patterns. The other simulation results support the 
generality of the retrieval ability in our model. Therefore, we confirmed that the network of 
oscillators proposed in this paper works well for associative memory of phase patterns. 

We would like to make some comments here before concluding. First, the synaptic 
efficacies, CV,-, was regarded as complex numbers. This may be naturally explained by the 



fact that the neurons are coupled via more than one component [[14]]. Secondly, even if the 
natural frequencies have a nontrivial distribution, it is expected that the retrieval of the 
patterns will be achieved by the network. By analogy with a network of phase oscillators, it 
may then be required that k assumes a value larger than a certain critical value k c . Finally, 
it is well known that the storage capacity of an oscillator network constructed by using the 
Hebbian rule is given by a c = P/N = 0.0377 [15-0]- Therefore, using the generalized 



Hebbian rule, we expect that our models work well when a < a c . 

In conclusion, we proposed a network of oscillators for the retrieval of phase information. 
We showed that this network has the ability to retrieve given patterns in which the oscillators 
keep a fixed phase relationship with one another. We would like to emphasize that, in the 
case of the second model, the network can retrieve patterns which include non-firing states. 
The amplitudes of the oscillators then play a crucial role in representing non-firing states. 
Furthermore, it is shown that under suitable conditions the system has a Lyapunov function 
ensuring a stable retrieval process. Using numerical simulations, we confirmed the good 
performance of our models. Consequently, we believe that the proposed models serve as a 
convenient starting point for the study of oscillatory neuronal systems. 



ACKNOWLEDGMENTS 

I would like to thank Y. Kuramoto, K. Okuda, T. Chawanya and I. Nishikawa for fruitful 
discussions and suggestions. I also thank G. Paquette for critically reading the manuscript. 
This work is supported by the Japanese Grant-in- Aid for Science Research Fund from the 
Ministry of Education, Science and Culture (Nos. 06260102 and 06740321). 



7 



REFERENCES 



[1] J.J. Hopfield, Proc. Nat. Acad. Sci. USA 79, 2554 (1982). 
[2] L.F. Abbot, J.Phys. A23, 3835 (1990). 
[3] T. Aoyagi, Europhys. Lett. 20, 565 (1992) . 

[4] Y. Kuramoto, T. Aoyagi, I. Nishikawa, T. Chawanya and K. Okuda, Prog. Theor. Phys. 
87, 1119 (1992). 

[5] CM. Gray, P. K6nig, A.K. Engel and W. Singer, Nature 338, 334 (1989). 

[6] J.J. Collins and I.N. Stewart, to appear in the Journal of Nonlinear Science. 

[7] P.CMatthews, R.E.Mirollo and S.H.Strogatz, Physica D52, 293 (1991). 

[8] T. Fukai, and M. Shiino, Europhys. Lett. 26, 647 (1994). 

[9] H. Sompolinsky, H. Golomb and D. kleinfeld, Phys. Rev. A43, 6990 (1991). 
[10] L.G. Personnaz, I. Guyon and G. Dreyfus, Phys. Rev. A34, 4217 (1986). , J. Physique 

Lett. 46, L359 (1985). 
[11] T. Kohonen, Self- Organization and Associative Memory (Springer- Verlag, 1984). 
[12] N. Nakagawa and Y. Kuramoto, Prog. Theor. Phys. 89, 313 (1993). 
[13] V. Hakim, and W. J. Rappel, Phys. Rev. A46, 46 (1992). 
[14] B. Ermentrout, and N. Kopell, Neural comp. 6, 225 (1994). 
[15] J. Cook, J. Phys. A22, 2057 (1989). 

[16] F. Gerl, K. Bauer, and U. Krey, Z. Phys. B88, 339 (1992). 
[17] K. Okuda, unpublished. 



8 



FIGURES 



FIG. 1. Typical temporal pattern of retrieval process in the second model. Black horizontal 
bars show active phases which are defined by RelFj > 0.5. The network succeeds in retrieving one 
of 8 memorized patterns. Note that the memorized pattern includes non-firing states. 

FIG. 2. Time development of eight overlaps in the same simulation as Figure 1. The solid line 
shows the overlap M\ concerning the retrieval pattern. 
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